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Abstract — In this paper, compressed sensing with noisy mea- 
surements is addressed. The theoretically optimal reconstruction 
error is studied by evaluating Tanaka's equation. The main 
contribution is to show that in several regions, which have 
different measurement rates and noise levels, the reconstruction 
error behaves differently. This paper also evaluates the per- 
formance of the belief propagation (BP) signal reconstruction 
method in the regions discovered. When the measurement rate 
and the noise level lie in a certain region, BP is suboptimal with 
respect to Tanaka's equation, and it may be possible to develop 
reconstruction algorithms with lower error in that region. 

Index Terms — Belief Propagation, Compressed Sensing, Noisy 
Signal Reconstruction, Tanaka's Equation. 

I. Introduction 

A. Motivation 

Compressed sensing (CS) is an emerging area of signal 
processing that allows to reconstruct sparse signals from a 
reduced number of measurements [1-3]. Because real world 
applications involve noisy measurements, CS with noise has 
drawn a lot of attention [4-7]. 

In recent work, Wu and Verdu [8] defined the noise sensitiv- 
ity as the ratio of the reconstruction error to the noise variance. 
The reconstruction is called robust if the noise sensitivity is 
finite. Wu and Verdu proved that the reconstruction is robust 
only when the measurement rate R, which is the ratio of the 
number of measurements to the signal length, exceeds a certain 
threshold. Wu and Verdu's threshold deals with the case where 
the noise is low. Unfortunately, the measurement noise in real- 
world applications might be greater than that required by Wu 
and Verdu. Thus, it is of interest to investigate the behavior 
of the CS reconstruction error in regions where there is more 
noise. 

B. Contribution 

Tanaka's fixed point equation provides the fundamental 
information theoretical limit on the reconstruction perfor- 
mance, which is quantified as the minimum mean square 
error (MMSE) of signal reconstruction in the presence of 
measurement noise [9-13]. In this paper, we use Tanaka's 
equation to evaluate the reconstruction performance for sparse 
Gaussian signals. That is, each element follows a Gaussian 
distribution with probability p, while it is zero with probability 
1 — p. We call p the sparsity rate of the signal. 

It might seem that lower noise always results in better recon- 
struction performance. However, the main result of this paper 
is to show that the behavior of the reconstruction performance 
is more nuanced in CS. There are several different performance 




Fig. 1. Different regions and thresholds in CS reconstruction. 

regions and thresholds that separate these regions, as illustrated 
in Figure 1. (See Section III for a detailed discussion. Note 
that the 7 in Figure 1 stands for the inverse noise level; larger 
7 means less noise.) 

• Low measurement region: The measurement rate R is 
too small. Robust reconstruction is impossible. 

• High measurement region: R is sufficiently large. This 
region consists of Region 1, and parts of Regions 4 and 5. 
Increasing the inverse noise level 7 leads to an immediate 
improvement in performance. 

• Medium measurement region: R is modest. This region 
consists of Regions 2 and 3, and parts of Regions 4 and 5. 
In the following we add further detail about the medium 
measurement region. 

1) Region 4: (Including the portion of Region 4 in the 
high measurement region.) The noise is high in this 
region, and thus, the reconstruction quality is poor 

2) Regions 2 and 3: The noise is modest in these 
regions. For fixed R, when the noise decreases (7 
increases) we leave Region 4 and enter Region 2. 
The performance hits a barrier and stays roughly 
constant. This barrier is broken through when we 
further increase 7 and enter Region 3, and the 
performance improves significantly. 

3) Region 5: (Including the portion of Region 5 in 



the high measurement region.) The noise is low 
in this region. As the noise decreases to zero, the 
reconstruction error vanishes. This is the case that 
Wu and Verdu studied [8]. 
We call the medium and high measurement regions the robust 
region, because the noise sensitivity within these regions is 
finite [8]; further insights are given in Section III.B. Of 
particular interest within the robust region is its portion where 
the noise is medium to high (Regions 1-4); we call this the 
realistic case. 

In this paper, we also discuss belief propagation (BP) [11, 
14-17], which is a signal reconstruction method that often 
achieves the theoretically optimal performance of CS in the 
large system limit. Having discussed the regions mentioned 
above, we evaluate the performance of BP in Section IV. 
Another key finding of this paper is that BP is suboptimal 
in Region 3. 

C. Organization 

The rest of this paper is organized as follows. Section II 
discusses background material including Tanaka's fixed point 
equation. In Section III, we describe the problem setting and 
expUcitly explain the different performance regions in detail. 
We then compare the performance of BP to the theoretically 
optimal performance of CS in Section IV. Section V concludes. 

II. Background 

We begin by defining some notations. Let x be the input 
signal, with each individual element being denoted hy Xi, i € 
{1,2, .... A^}. The input x is acquired by multiplying it by a 
measurement matrix $ G M*^^^, which has unit norm rows. 
Our model also involves additive measurement noise, which is 
denoted by z. The input signal x, the channel noise z, and the 
entries of $ are independent and identically distributed (i.i.d.), 
respectively. The measurement vector y e is given by 



y = s/l^x + z, 



(1) 



where 7 is the ampUfication of the channel, which represents 
the inverse noise level (1) and thus relates to the signal-to- 
noise ratio (SNR). The number of measurements, which is 
also the length of y, is denoted by M. Finally, we define the 
measurement rate as 



R 



M 
~N' 



Guo and Wang [11] stated that the fundamental informa- 
tion theoretical characterization of the optimal reconstruction 
performance in CS remains the same when using a fixed R, 
where M and scale to infinity, i.e., limjv^-oo ^ = -R; this is 
called the large system limit. We discuss signal reconstruction 
performance as a function of the measurement rate R and the 
inverse noise level 7 in the large system limit. 

According to Guo and Wang [11], the problem of estimating 
each scalar Xj given the vector y in the vector channel (1) 
is asymptotically equivalent to estimating the same symbol 
through a scalar channel. 



Vi = \J VlRxi + Zi, 



where yi and Zi, i G {1,2. ...,7V}, are the sufficient statistics 
for y and z in the vector channel (1), respectively, and 
T] e (0, 1) is the degradation of the scalar channel relative 
to the original vector channel. The amount of noise in (2) can 
be computed by finding a fixed point for 77 in the following 
equation by Tanaka [8, 9], 

i = 1 + 7 • MMSE, (3) 
V 

where MMSE is the minimum mean square error of the esti- 
mation in the equivalent scalar channel (2). Tanaka's equation 
may give more than one fixed point (an example for multiple 
fixed points can be found in [8]); the correct fixed point 
minimizes the free energy [12, 13], 



R. 



E{f]) = I{x,; ^/rryRxi + z^) + — (77 - 



l-log2(r,)), (4) 



where /(•; •) is the mutual information [18]. Note that Tanaka's 
results are based on the replica method in statistical physics, 
and are not rigorous [10]. On the other hand, replica analyses 
in compressed sensing can be made rigorous in some cases; 
see Bayati and Montanari [19] for example. 

By evaluating (3) and (4), we can compute the degradation 
77 of the scalar channel, as well as the MMSE. With this 
MMSE, we can determine whether a certain measurement rate 
R provides acceptable reconstruction performance. Note that 
Tanaka's equation can also yield information theoretical limits 
on the lowest attainable mean error when non-quadratic error 
metrics are considered |20|. 

III. Reconstruction in the robust region 

Having discussed Tanaka's equation, we now employ it in 
analyzing the sparse Gaussian case, with the i.i.d. input signal 
X following the distribution function. 



fXi{Xi) = p- 



1 



V27r 



+ {I - p) ■ 5a{xi) 



(5) 



for i G {1, N}, where 5a{x) denotes the delta function with 
probability mass at the origin. We further assume that the noise 
z is i.i.d. unit variance zero mean Gaussian distributed. 



1 



27r 



(6) 



In the sparse Gaussian case, SNR= 7 • p for the vector 
channel (1), whereas SNR= 777P for the scalar channel (2). 

Tanaka's equation (3) allows us to investigate the vector 
problem (1) by analyzing the corresponding scalar channel (2). 
It can be shown that in the scalar channel (2), the MMSE of 
the sparse Gaussian case satisfies 



MMSE = p ■ 



p^aR 
27r(aii 



(7) 



y p.e2(aB.+i)y + (l ~p)y/aR+l ' e^foK+D^ 



:dy, 



(2) 



where a = i^y for notational simplicity. 

After specifying the sparsity rate p, the inverse noise level 
7, and the measurement rate R, we obtain the channel degra- 
dation T] and thus the MMSE in (3). In the following two 



subsections, we first discuss the reconstruction performance 
in the low noise case, i.e.. Region 5, and then discuss the 
realistic case, which includes Regions 1-4. 

A. CS reconstruction performance in the low noise case 

Wu and Verdu [8] analyzed the robustness of reconstruction 
performance (MMSE) in the low noise region (Region 5) 
where 7 — >■ oo, 



MMSE(p, r77i?) 



P 



(1 + 0(1)), 



(8) 



where the o(l) term vanishes as 7 — )• 00. Substituting (8) into 
(3) and solving for 77, we obtain 



1 



0(1)) 



and 



MMSE(j9, rj-yR) = 



liR-p) 



(1 + 0(1)). 



Wu and Verdti [8] define the noise sensitivity as MMSE 
divided by the noise variance. In this case the noise sensitivity 



MMSE(p, rjjR) p 



1/7 



R-p 



(1 + 0(1)) 



is finite only when R > p. Therefore, the threshold for robust 
reconstruction is R = p for sparse Gaussian signals. 

As the reconstruction is not robust below the threshold, i.e., 
R < p, we are only interested in the robust region, R > p. 
Also, considering that the measurement rate in a CS system 
is lower than 1, we concentrate on the case when p < R < 1. 
We can see that the MMSE (7) is a function of a = rj'y. If 7 
grows and a stays constant for a fixed R, then we will obtain 
a constant MMSE and an ry that decreases to zero. As a result, 
(3) can be rewritten as 



1 - a • MMSE(p, aR) 



rj > 0. 



(9) 



That is, the fixed point of the form -q — - satisfies (3). 
Therefore, when solving Tanaka's equation in Region 5, there 
could be several fixed points: («.) the first fixed point is 
771 = where ai is a constant for a fixed R\ (ii.) the second 
is 772 = where a2 is another constant, which is larger than 
fli, for a fixed R; and (///.) the third is 773 — 1 — J^- The correct 
fixed point minimizes the free energy (4) and is determined 
later. 

For R > i?bp(7), there is only one fixed point % = 1 — |j 
and there is no need to evaluate the free energy. At Rbp{j), 
there are exactly two fixed points, which are r/i — ^ and 
?73 = 1 — ^. When reducing R below the threshold Rbp{-y), 
r]2 appears. Note that in the realistic case with more noise, the 
approximations above for 771, 772, and 7/3 become imprecise; 
the discussion for this case appears in Section lll.B. 

If the optimal performance is obtained by 771 or 772, then the 
SNR of the corresponding scalar channel remains constant as 
the noise declines, and the channel ends up with a roughly 
fixed reconstruction error. However, because a2 is larger than 
Si, the SNR will be greater in the case of 772 than that of 771. 
Thus, if 771 = Y snds up being the correct fixed point, then the 
performance will be the worst among the three fixed points. 
Therefore, we prefer the case that % = 1 — ^ minimizes the 



free energy, because this fixed point 773 leads to the lowest 
MMSE. 

We now evaluate the free energy of each fixed point. The 
one that minimizes the free energy (4) corresponds to the 
theoretically optimal performance of the channel. As we have 
already seen that in Region 5 we could have r]i = ^,ri2 = 
and 773 = 1 — ^, we need to evaluate the free energy of each 
fixed point. 

The mutual information of the channel (2) can be approxi- 
mated as in [21], 



I{xi\yi) ~ -p\ia.{r]^R). 



Plugging (10) into (4), for 7/1 = ^i. 



(10) 
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^ln(SO + fln(i?)+ 



i? „ R R 
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(11) 



for 7?2 = f , 



R 
ln(2) 



ln(a2) 



iog2(7); 



|ln(i?)- 



R ^ R R , J- ^ 
-a2 - 2 + 2l°S2(7); 



(12) 



and for 773 



p_ 

R' 



^'~Er(2), p. , p, ,R. , / X .10. 
2 R^ 2^^~^^ 2^^'^^' 



Considering that p < R < 1, we have 



Eim) 



R 



R 



log2(7), ^(^72) ~ 771082(7) 



and -£(773) ~ - ln(7) 

for large 7. The only difference between (11) and (12) is in 
the constants ai and 0.2- Provided that ai < 0,2 and R > p, 
we obtain that -©(773) < i?(77i) < -£(772). Thus, the correct 
7? is 773 = 1 - and MMSE(p,777i?) = ^^^(l + o{l)) 
decreases to zero as 7 increases to 00. That is, in the robust 
region the reconstruction error vanishes as long as 7 is 
sufficiently large (Region 5). 

B. CS reconstruction performance in the realistic case 

The MMSE can be driven down to zero in the robust region 
by driving down the noise to zero. However, we cannot use an 
arbitrarily large 7, owing to the physical characteristics of the 
measurement system. Thus, it is worthy to study the reahstic 
case (Regions 1-4) where the noise is not small. 

In the realistic case, the approximation of the fixed points 
might be imprecise. Also, the approximation of the free 
energy (11) - (13) does not apply, because 7 does not go 
to 00. Therefore, we must calculate each fixed point and the 
corresponding free energy (4). Evaluating Tanaka's equation 
numerically gives Figure 1, which appeared in Section I. The 
following discussion explains the figure in detail. 



1) Above the BP threshold Rtpij), BP is advantageous 
(Section IV). Below i?bp(7), the MSE performance of 
BP is not satisfactory. 

2) The threshold that separates the robust region and the 
low measurement (unrobust) region is called the robust 
threshold; it is denoted by Rr, and was characterized 
comprehensively by Wu and Verdii [8]. 

3) In Regions 1 and 4 there is only one fixed point; these 
two regions merge together as R is increased. The 
difference is that the SNR in Region 1 is higher, thus 
the reconstruction error in Region 1 is lower than that 
in Region 4. 

4) The boundary separating Region 2 and 4 is the consis- 
tency threshold Rd'j), beyond which the reconstruction 
error stays roughly consistent for a fixed R in Region 2, 
because the smallest 771 « ^ minimizes the free energy, 
and the scalar channel SNR= rj'yp is roughly constant. 

5) The boundaries separating Regions 1 and 4 from Re- 
gions 2 and 3 are the consistency threshold Rc{l) 
and the BP threshold Ri,p{j); they are the thresholds 
bounding Regions 2 and 3, where three fixed points 
emerge. On Rc{j) and Ri,p{-f), there are exactly two 
fixed points. 

6) Regions 2 and 3 are separated by the low noise thresh- 
old Rii'y), which means that above this threshold, the 
MMSE behaves as in the low noise case. In Region 3, 
which lies between the thresholds Ri{'j) and Ri,p{j), 
the reconstruction error decreases as 7 increases for a 
fixed R, because the biggest rj (not necessarily equal to 
1 — ^) minimizes the free energy. 

As we have seen, in Region 2 the MMSE is roughly fixed 
for a certain R. On the other hand, in Regions 1 and 3 the 
reconstruction error decreases as 7 is increased. However, the 
boundary between Regions 2 and 3, Rii'y), approaches R,- as 
7 increases. Therefore, for a fixed measurement rate greater 
than Rr, if we increase 7, then we will eventually traverse 
Riiy) and obtain a significant reduction in the reconstruction 
error as we move from Region 2 to Region 3. 

By specifying the sparse Gaussian problem as p = 0.1 and 
R e (0.12,0.23), we obtain the MMSE, which is shown in 
Figure 2; Regions 1-4, as well as the thresholds Rbpij), Rdl), 
and Rii'y), are also marked out in the figure. 

We can see in Figure 2 that for a relatively small R, when 
7 increases, the MMSE first decreases slightly, then it remains 
roughly constant, and after a further increase in 7 the MMSE 
experiences a significant reduction and vanishes. On the other 
hand, for large R, increasing 7 always leads to a prominent 
decrease in the MMSE. 

IV. Performance of Belief Propagation 

Having characterized the different performance regions, we 
aim in this section to shed light on the performance regions 
that practitioners may encounter when implementing CS re- 
construction algorithms in real systems. Belief propagation 
(BP) [15] is known to achieve the MMSE performance of 
the smallest fixed point, and it is optimal for many problems 
of interest. Thus, it is interesting to evaluate the performance 
of BP on the regions that were discussed in Section III. 




Fig. 2. Theoretical CS performance of a sparse Gaussian signal (p = 0.1) 
provided by Tanaka's equation. Regions 1-4, as well as the thresholds Rbpi'j), 
Rii'y), and Rci'y), are marked out. 



We used a BP solver, GAMP [17], to simulate a se- 
ries of CS reconstruction problems. The cases where R G 
{0.15, 0.16,. ..,0.23} and 7 e {lOdB, 15dB, .., 70dB} were 
simulated. For each R and 7, 100 sparse Gaussian signals 
with a length of 10,000 and sparsity rate p = 0.1, as well 
as the corresponding measurement matrices $ and noise z, 
were generated. The matrices were i.i.d. Gaussian, with zero 
mean and unit norm rows; the noise followed Af{0, 1). We 
averaged the empirical MSE's for each of the 100 signals. The 
simulation results, as well as the theoretical MMSE given by 
Tanaka's equation, are illustrated in Figure 3. 

We can see that GAMP results in a reconstruction error that 
is close to the MMSE in the regions where there is only one 
fixed point, which includes Region 1 (corresponds to i? > 
0.21 in Figure 3) and Region 4 (corresponds to 7 roughly less 
than 40 dB). However, in the region where there are several 
fixed points, GAMP operates at the smallest fixed point, as 
discussed by Guo and Wang [14] and Rangan [15]. That is, BP 
is optimal in Regions 1, 2 and 4, while suboptimal in Region 3. 
When 7 becomes large (for fixed R), we will inadvertently 
leave Region 2 and enter Region 3, where BP is suboptimal. 
The performance deficiency of BP is also the reason why we 
denote the boundary between Regions 1 and 3 by R^piy); it 
is the threshold above which BP is advantageous. Below the 
threshold, the MSE performance of BP is not satisfactory. For 
Region 3, which is part of the region having modest 7 and 
R, it might be possible to develop new algorithms that have 
better performance than BP. 

From Figure 2, it appears that Rbp{y) converges to some 
Rbp in the Umit of large 7; more work is being done to 
demonstrate this convergence rigorously. We can approximate 
Rbp for different p by evaluating Tanaka's equation for some 
reasonably large 7. The result is shown in Figure 4. 

V. Conclusion 

In this paper, we discussed the reconstruction performance 
of sparse Gaussian signals. By evaluating Tanaka's fixed point 
equation (3) and the free energy (4), we found that there 
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Fig. 4. Sparsity rate p v.s. the threshold for BP to be advantageous 



are several regions where the reconstruction error behaves 
differently. We further compared the reconstruction perfor- 
mance of BP to the theoretically optimal performance, and 
showed that this state-of-the-art method has its limitations. 
Based on this observation, there is room for a new generation 
of reconstruction algorithms with better performance than BP 
in the region where both the inverse noise level and the 
measurement rate are modest. 
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